Hadoop Perfect File: A fast and memory-efficient metadata access archive file to face small files problem in HDFS

HDFS faces several issues when it comes to handling a large number of small files. These are well addressed by archive systems, which combine files into larger ones. They use index hold relevant information for retrieving file content from the big file. However, existing archive-based solutions require significant overheads since additional processing and I/Os needed acquire retrieval before accessing actual content, therefore, deteriorating access efficiency. This paper presents new named Hadoop Perfect File (HPF). HPF minimizes directly metadata part containing information. It consequently reduces improves efficiency Our system uses two hash functions. Metadata records distributed across using dynamic function. We further build an order-preserving perfect function that memorizes position file's record within